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Abstract 


Since  2001 ,  12  major  defense  acquisition  programs  (MDAPs)  have  been 
cancelled.  Although  each  of  these  programs  had  problems  with  cost  or  schedule 
overruns  (or  both),  there  were  other  MDAPs  that  had  similar  problems  and  were  not 
cancelled.  Is  it  possible  that  program  managers  had  information  that  might  help 
determine  which  program  was  likely  to  survive  and  which  was  more  likely  to  be 
cancelled?  We  employ  a  unique  and  rigorous  statistical  methodology  to  help 
program  managers  and  their  overseers  understand  and  quantify  the  risk  to  their 
programs  based  on  key  earned  value  metrics.  We  compare  programs  that  were 
cancelled  to  programs  that  had  significant  cost  overruns  but  were  not  cancelled.  We 
use  survival  analysis  to  investigate  whether  differences  in  key  EV  metrics  reported 
for  cancelled  programs  and  “troubled”  but  not  cancelled  programs  can  be  used  to 
model  the  probability  of  cancellation  for  MDAPs.  Our  most  significant  finding  across 
models  is  that  when  there  is  high  cost  growth  in  the  EAC  reported  by  the  contractor, 
programs  run  far  larger  risks  of  cancellation.  We  find  less  robust  evidence  that 
increases  in  PM  estimates  and  high  cost  variance  also  can  drive  risk  of  program 
cancellation. 

Keywords:  Acquisition,  Earned  Value,  Survival  Analysis,  Cost  Growth. 
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Using  Earned  Value  Information  to  Predict 
Program  Cancellation 


Introduction 

Since  2001 ,  12  major  defense  acquisition  programs  (MDAPs)  have  been 
cancelled.  Although  each  of  these  programs  had  problems  with  cost  or  schedule 
overruns  (or  both),  there  were  other  MDAPs  that  had  similar  problems  and  were  not 
cancelled.  Is  it  possible  that  program  managers  had  information  that  might  help 
determine  which  program  was  likely  to  survive  and  which  was  more  likely  to  be 
cancelled? 

MDAPs  are  programs  designated  by  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  (USD  [AT&L])  as  having  an  estimated 
eventual  total  expenditure  for  Research,  Development,  Test  and  Evaluation 
(RDT&E)  of  more  than  $365  million  in  fiscal  year  (FY)  2000  constant  dollars  or,  for 
procurement,  of  more  than  $2.19  billion  in  FY  2000  constant  dollars  (Defense 
Acquisition  University,  2012a). 

Information  about  cost  and  schedule  performance  for  MDAPs  is  provided  to 
DoD  using  Earned  Value  Management  (EVM).  EVM  is  an  integrated  management 
approach  that  uses  schedule,  cost,  and  scope  of  work  goals  and  measures 
progress  towards  achievement  of  these  goals  (Performance  Assessments  and  Root 
Cause  Analyses,  2013).  It  provides  program  management  offices  (PMOs)  with 
information  to  measure  progress  against  established  baselines  and  is  required  for  all 
MDAPs. 

This  paper  employs  unique  and  rigorous  statistical  methodology  to  help 
program  managers  and  their  overseers  understand  and  quantify  the  risk  to  their 
programs  based  on  key  earned  value  metrics.  It  compares  programs  that  were 
cancelled  to  programs  with  that  had  significant  cost  overruns  but  were  not  cancelled. 
We  use  survival  analysis  to  investigate  whether  differences  in  key  EV  metrics 
reported  for  cancelled  programs  and  “troubled”  but  not  cancelled  programs  can  be 
used  to  model  the  probability  of  cancellation  for  major  acquisition  programs. 

Specifically,  we  look  at  unclassified  MDAPs  cancelled  since  2001  and  similar 
programs  that  could  be  described  as  “troubled”  but  not  cancelled  in  the  same  period 
of  time.  MDAPs  were  chosen  because  they  are  statutorily  required  to  report  EV  data. 
We  use  the  EV  data  reported  in  the  Defense  Acquisition  Executive  Summaries 
(DAES)  on  the  government’s  Defense  Acquisition  Management  Information  Retrieval 
(DAMIR)  website. 
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Literature  Review 


Most  of  the  studies  that  have  examined  why  MDAPs  fail  point  to  cost 
overruns  as  the  primary  cause.  Reasons  cited  include  (1)  excessively  low  initial 
cost  estimates  (Government  Accountability  Office,  2009;  Sipple,  White,  &  Greiner, 
2004),  (2)  expectation  of  cost  overrun  sharing  causing  firms  to  bid  below  cost 
estimate  (Chen  &  Smith,  2001),  (3)  excessively  low  cost  overrun  estimates 
(Christensen,  1994),  (4)  technological  immaturity  (Dubes,  Saleh,  &  Braun,  2007; 
Tyson,  Harmon,  &  Utech,  1994),  (5)  unstable  requirements  (Augustine,  1997),  and 
(6)  overly  optimistic  schedules  (Augustine  1997;  Berteau  et  al.,  2011). 

Charette  (2008)  pointed  out  that  defense  acquisitions  problems  have  existed 
for  decades,  but  the  economic  scope  has  changed.  To  put  the  scope  of  defense 
spending  in  context,  the  DoD’s  2013  portfolio  of  80  MDAPs  has  a  total  estimated 
cost  of  $1 .5  trillion  (Government  Accountability  Office,  201 4);  Berteau  et  al.  (201 1 ) 
investigated  the  root  cause  of  cost  and  schedule  overruns  and  found  that  inaccurate 
cost  estimates  are  associated  with  40%  of  the  accumulated  cost  overruns. 

Many  authors  advocate  the  use  of  EVM  as  a  tool  to  help  program  managers 
and  decision  makers  monitor  early  signs  of  cost  growth  (see,  for  example,  Kerzner, 
2009  and  Webb,  2003).  Abba  (1997)  made  it  clear  that  the  U.S.  government  and 
countries  across  the  world  have  validated  EVM  as  a  highly  effective  program 
management  tool.  Christensen  (1999)  noted  that  EV  provides  program  managers 
and  contractors  valuable  insight  into  the  cost  and  schedule  status  of  the  project 
and  concluded  through  multiple  studies  that  the  estimate  at  completion  (EAC)  is 
one  of  the  most  critical  values  reported  to  PMs. 

Extensive  research  and  experience  have  convinced  both  Congress  and 
USD(AT&L)  to  conclude  that  EVM  is  a  highly  effective  program  management  tool 
(Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  201 1 ; 
Weapon  Systems  Acquisition  Reform  Act,  2009).  The  Weapon  Systems  Acquisition 
Reform  Act  of  2009  created  the  Director,  Performance  Assessment  and  Root 
Cause  Analysis  (PARCA)  (Weapon  Systems  Acquisition  Reform  Act,  2009).  USD 
AT&L  has  made  PARCA  responsible  for  EVM  performance,  oversight,  and 
governance  and  for  leading  EVM  improvements  across  the  DoD  (Under  Secretary 
of  Defense  for  Acquisition,  Technology,  and  Logistics,  2011). 

Numerous  factors  are  responsible  for  program  cancellation.  While  many 
programs  are  troubled,  not  all  of  them  are  cancelled.  This  paper  investigates 
whether  there  are  significant  differences  in  Earned  Value  metrics  between  cancelled 
and  non-cancelled  programs,  and  whether  these  metrics  can  help  predict  the 
ultimate  cancellation  and  timing  of  cancellation  of  programs.  This  paper 
hypothesizes  that  cancelled  programs  would  have  more  unfavorable  cumulative  cost 
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and  schedule  variances,  greater  cost  growth  in  the  estimate  at  completion  (EAC), 
and  more  disparity  between  the  contractor  and  program  manager  cost  estimates. 

Earned  Value  Management 

Earned  Value  Management  is  a  widely  accepted  industry  best  practice  that  is 
used  commercially  and  in  the  DoD  to  manage  programs.  It  is  an  integrated 
management  approach  that  uses  schedule,  cost,  and  scope  of  work  goals  and 
measures  progress  towards  achievement  of  these  goals  (Performance  Assessments 
and  Root  Cause  Analyses,  201 3).  It  is  a  critical  tool  for  engineering  management 
and  oversight  of  acquisition  (Defense  Acquisition  University,  2012a). 

In  simplest  terms,  EVM  is  a  procedure  for  understanding,  assessing,  and 
quantifying  what  a  contractor  is  achieving  with  contract  dollars  and  to  predict  future 
performance.  It  works  by  establishing  an  integrated  baseline  that  is  developed  from 
the  work  defined  in  the  work  breakdown  structure  and  its  associated  time-phased 
budget.  As  work  is  performed,  its  corresponding  budget  (“earned  value”)  can  be 
measured  against  the  integrated  baseline.  Cost  and  schedule  variances  can  be 
calculated  and  analyzed.  These  variances  can  help  management  determine  if  a 
project  is  ahead  or  behind  schedule  and  above  or  below  budget,  and  where  to  focus 
additional  resources  to  remedy  the  problem. 

EVM  Statutes,  Policy,  and  DoD  Implementation 

EVM  evolved  during  the  1990s  from  service  unique  cost  and  schedule 
performance  criteria  called  the  Cost/Schedule  Control  System  Criteria  (C/SCSC) 
into  a  set  of  32  industry-owned  guidelines  called  the  Earned  Value  Management 
System  (EVMS;  Defense  Acquisition  University,  2012b).  The  requirement  for 
MDAPs  to  use  EVM  is  stipulated  in  three  laws: 

•  Government  Performance  and  Results  Act  of  1 993  (GPRA) 

•  Federal  Acquisition  Streamlining  Act  of  1 994  Title  V  (FASA) 

•  Clinger-Cohen  Act  of  1 996 

These  acts  provide  the  legal  basis  for  the  policies  implementing  EVM.  At  the 
executive  branch  level,  the  primary  policy  document  governing  EVM  is  the  Office  of 
Management  and  Budget  (0MB)  Circular  A-1 1 ,  Part  7,  Capital  Programming  Guide, 
which  requires  the  use  of  EVM  or  some  similar  system  “for  risk  and  program 
management  of  capital  asset  acquisition”  and  “to  establish  cost,  schedule,  and 
performance  goals  for  major  acquisitions  and  then  achieve  on  average,  90%  of 
these  goals”  (Office  of  Management  and  Budget,  2006).  The  DoD,  as  an  agency  of 
the  federal  government,  has  issued  its  own  directives — the  DoDD  5000.01  (Office  of 
the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  &  Logistics 
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[OUSD(AT&L)],  2007)  and  DoDI  5000.02  (OUSD[AT&L],  2008)— to  address  the 
statutory  and  regulatory  requirements  of  acquisition  of  all  military  systems.  These 
documents  along  with  the  Federal  Acquisitions  Regulation  (FARs)  and  the  Defense 
Federal  Acquisitions  Regulation  (DFARs)  provide  the  explicit  requirements Mor  use 
of  EVM  in  DoD  MDAPs. 

The  Weapon  Systems  Reform  Act  of  2009  created  the  PARCA  and  charged 
the  organization  with  conducting  and  overseeing  performance  assessments  and  root 
cause  analyses  for  MDAPs  (Weapon  System  Acquisition  Reform  Act,  2009). 
PARCA’s  assessments  evaluate  the  cost,  schedule,  and  performance  requirements 
relative  to  current  metrics  (Defense  Acquisition  University  2012a).  Additionally, 
PARCA  is  the  policy  holder  for  EV;  they  are  responsible  for  the  implementation  and 
use  of  EVM  across  the  DoD  and  for  evaluating  the  utility  of  performance  metrics 
used  to  measure  cost,  schedule,  and  performance  of  MDAPs  (Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics,  201 1 ). 

In  a  recent  DoD  memorandum  entitled  “Earned  Value  Management  Systems 
Performance,  Oversight,  and  Governance,”  the  USD(AT&L)  provided  direction  to 
improve  the  effectiveness  of  EV  across  the  DoD.  Specifically,  he  reemphasized  that 
EVM  must  be  applied  in  a  disciplined  manner  and  that  the  data  provided  by  EVM 
must  be  accurate,  reliable  and  timely  (Under  Secretary  of  Defense  for  Acquisition, 
Technology,  and  Logistics,  2011).  The  Defense  Contract  Management  Agency’s 
(DCMA)  is  responsible  for  EVM  System  compliance  within  the  DoD.  (Under 
Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics,  2011).  In  this  role, 
DCMA  conducts  EVMS  reviews  of  all  MDAPs  to  ensure  compliance  of  EV 
standards.  The  DCMA’s  Earned  Value  Management  Implementation  Guide  (EVMIG) 
is  the  DoD’s  principle  guidance  document  for  EV. 

EVM  Reporting  Requirements 

The  three  primary  vehicles  for  reporting  EVM  information  are  (1)  Selected 
Acquisition  Reports  (SARs)  to  Congress,^  (2)  Defense  Acquisition  Executive 
Summary  (DAES)^  reports  to  senior  level  DoD  decision-makers^  and  (3)  the 


^  Current  DoD  regulation  and  policy  requires  EVMS  on  cost  or  incentive  contracts,  subcontracts,  and 
intra-  government  work  agreements  valued  at  or  greater  than  20  million  in  then-year  dollars.  For 
efforts  exceeding  50  million  in  then-year  dollars,  the  EVMS  must  be  validated  or  accepted  by  the 
Defense  Contract  Management  Agency  (Defense  Acquisition  University  2012a). 

^  DoD  must  submit  Selected  Acquisition  Reports  (SARs)  for  all  MDAPs  annually.  The  frequency  is 
increased  should  the  MDAP  fail  to  achieve  certain  performance  thresholds  contained  in  10  USC  § 
2432 — Selected  Acquisition  Reports.  The  SARs  enable  USD(AT&L)  to  meet  statutory  reporting 
requirements  of  all  MDAPs  to  Congress  (Defense  Acquisition  University  2012a). 

^  DAESs  are  submitted  quarterly  or  monthly  depending  on  whether  certain  performance  thresholds 
are  met.  The  DAES  process  enables  the  USD(AT&L)  to  fulfill  statutory  requirements  to  manage  and 
oversee  MDAPs.  The  goal  of  the  DAES  process  is  to  facilitate  communication  between,  and  provide 
feedback  to,  key  stakeholders  in  OSD,  the  Joint  Staff,  the  Components,  and  Program  Offices.  It  is 
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Integrated  Program  Management  Report  (IPMR)®to  program  managers.  The  goal  of 
all  of  these  reports  is  to  facilitate  communication  between  and  provide  feedback  to 
key  stakeholders  in  Congress,  DoD,  and  the  program  offices.  Typically,  the  EV  data 
for  MDAPs  is  reported  monthly  by  the  contractor  in  Contract  Performance  Reports 
(formats  1  through  5)®  and  in  the  IPMR  for  use  by  internal  program  management.  A 
portion  of  this  data  is  included  monthly  in  the  DAES  database,  quarterly  in  the  formal 
DAES  reports  for  use  by  DoD  executives,  and  annually  in  the  SARs  for  use  by 
Congress.^  The  DAES  reports  are  the  source  documents  for  EV  data  in  this 
research. 

Earned  Value  Terminology 

The  following  definitions  are  necessary  to  understand  the  analysis  and 
findings  of  this  report.  The  definitions  come  from  the  EVMIG  (Defense  Contract 
Management  Agency  2006)  and  the  Defense  Acquisition  University  Glossary  of 
Defense  Acquisition  and  Terms  (DAU,  2009).® 

•  Budgeted  Cost  of  Work  Scheduled  (BCWS  or  “planned  value”):  the 
sum  of  the  budgets  for  all  work  scheduled  to  be  accomplished  with  a 
given  time  period.  Also  called  the  Performance  Measurement  Baseline 
(PMB).  BCWScum  represents  the  cumulative  BCWS  at  a  certain  point 
of  the  contract. 

•  Actual  Cost  of  Work  Performed  (ACWP  or  “actual  costs”):  the  costs 
actually  incurred  and  recorded  in  accomplishing  the  work  performed 
within  a  given  time  period.  ACWPcum  represents  the  cumulative  ACWP 
at  a  certain  point  of  the  contract. 

•  Budgeted  Cost  of  Work  Performed  (BCWP  or  “earned  value”):  the 
value  of  completed  work  in  terms  of  the  work’s  assigned  budget. 
BCWPCUM  represents  the  cumulative  BCWP  at  a  certain  point  of  the 
contract. 


important  to  note  that  the  DAES  is  an  internai  management  system  meant  to  fuifiii  the  needs  of  senior 
Department  of  Defense  executives  and  is  NOT  for  generai  pubiic  consumption  (Defense  Acquisition 
University  2012a). 

For  the  remainder  of  this  study  the  generai  coiiective  term  “DoD  executives,”  “DoD  officiais,”  or 
“senior  ievei  DoD  decision  makers”  refers  to  USD(AT&L),  Program  Executive  Office  (PEO),  the 
Miiestone  Decision  Authority  (MDA),  and  their  associated  staffs. 

^The  IPMR  provides  performance  data  that  is  used  to  identify  probiems  eariy  in  the  contract  and 
forecast  future  contract  performance  (Defense  Acquisition  University,  2012a). 

®  Contract  Performance  Reports  formats  1  through  5  are  prepared  by  the  contractor  and  are  the 
primary  means  for  reporting  contract  performance  data.  Their  periodicity  is  typicaiiy  monthiy  uniess 
taiiored  for  specific  program. 

These  periodicities  are  subject  to  change  based  on  the  program. 

®  Except  where  specific  citation  is  made  in  this  section,  it  is  to  be  assumed  that  the  definition  came 
from  one  of  these  two  sources. 
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Schedule  Variance  (SV):  the  algebraic  difference  between  earned 
value  and  the  budget  (SV  =  BCWP  -  BCWS).  A  positive  value  is  a 
favorable  condition  (ahead  of  schedule)  while  a  negative  value  is 
unfavorable  (behind  schedule). 

Schedule  Variance  Percentage  (SV%):  indicates  how  much  ahead  or 
behind  schedule  the  project  is  in  terms  of  percentage.  A  positive  value 
is  a  favorable  condition  (percent  ahead  of  schedule)  while  a  negative 
value  is  unfavorable  (percent  behind  schedule).  It  may  be  expressed 
as  a  value  for  aspecific  period  of  time  or  for  cumulative  to  date.  SV%  = 
(SV/BCWS)  X  100  (TutorialsPoint,  2013). 

Schedule  Performance  Index  (SPI):  EV  performance  factor 
representing  schedule  efficiency.  Calculated  by  dividing  the  Budgeted 
Cost  for  Work  Performed  (BCWP)  by  the  Budgeted  Cost  for  Work 
Scheduled  (BCWS).  This  metric  is  one  of  the  performance  factors  used 
in  EAC  calculations. 

Cost  Variance  (CV):  the  algebraic  difference  between  earned  value 
and  actual  cost  (CV  =  BCWP  -  ACWP).  A  positive  value  indicates  a 
favorable  condition  (under  budget)  and  a  negative  value  indicates  an 
unfavorable  condition  (over  budget). 

Cost  Variance  Percentage  (CV%):  indicates  how  much  over  or  under 
budget  the  project  is  in  terms  of  percentage.  It  indicates  how  much  less 
or  more  money  has  been  used  to  complete  the  work  as  planned  in 
terms  of  percentage.  A  positive  value  is  a  favorable  condition  (percent 
under  budget)  while  a  negative  value  is  unfavorable  (percent  over 
budget).  It  may  be  expressed  as  a  value  for  a  specific  period  of  time  or 
for  cumulative  to  date.  CV  %  =  (CV/BCWP)  x  100  (TutorialsPoint, 
2013). 

Cost  Performance  Index  (CPI):  EV  performance  factor  representing 
cost  efficiency.  Calculated  by  dividing  the  Budgeted  Cost  for  Work 
Performed  (BCWP)  by  the  Actual  Cost  of  Work  Performed.  This  metric 
is  one  of  the  performance  factors  used  in  EAC  calculations. 

Budget  at  Completion  (BAC):  The  sum  of  all  budgets  established  for 
the  contract.  BAC  is  a  term  that  may  also  be  applied  to  lower  levels, 
such  as  the  PMB  or  at  the  control  account  level. 

Estimate  at  Completion  (EAC):  the  estimated  total  cost  for  all 
authorized  work.  Equal  to  the  sum  of  actual  costs  to  date  (including  all 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


allocable  indirect  costs),  plus  the  estimated  costs  to  completion 
(estimate  to  complete). 

•  Estimate  to  Complete  (ETC):  estimate  of  costs  to  complete  all  work 
from  a  given  point  in  time  to  the  end  of  the  contract. 

•  Variance  reset:  when  a  contract’s  cost  and/or  schedule  variances  are 
reset  to  zero.  This  is  done  to  improve  managerial  control  over  the 
work  remaining  on  a  contract. 

When  the  BCWS,  ACWP,  and  BCWP  are  obtained  for  a  period,  numerous 
additional  EV  metrics  can  be  calculated  including  the  SPI,  SV%,  CPI,  CV%,  ETC, 
and  EAC,  that  are  helpful  to  understand  how  the  program  is  performing  and  to 
predict  future  performance  based  on  the  contract’s  past  performance.  Figure  1 
illustrates  the  relationship  of  BCWS,  ACWP,  and  BCWP  for  a  project  that  is  over¬ 
budget  (ACWP>BCWP)  and  behind  schedule  (BCWP<BCWS). 


Figure  1.  Graphical  Depiction  of  EV  Metrics  for  a  Sample  Project 

(Vargas,  2009) 
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Calculating  EAC 

EAC  is  used  to  determine  whether  sufficient  funds  are  available  to  cover  the 
cost  of  the  contract  at  completion.  A  common  way  for  programs  to  calculate  their 
EAC  is  by  using  formulas  that  use  contractor’s  efficiency  to  date  as  measured  by  the 
CPI  and  SPI.  Equation  1  illustrates  that  EAC  is  equal  to  the  amount  of  money 
already  spent  on  the  contract  or  Actual  Cost  of  Work  Performed  (ACWP)  plus  the 
amount  of  money  it  will  take  to  complete  the  contract  or  the  Estimate  to  Complete 
(ETC). 

EAC  ^  ACWP  +  ETC  (1) 


The  ACWP  is  an  accounting  figure  reported  monthly  by  the  contractor  in  the  CPRs. 
The  ETC  on  the  other  hand  is  a  forecast  that  can  be  calculated  in  numerous  ways. 
The  generic  formula  for  calculating  ETC  is  contained  in  Equation  2: 


ETC  = 


BAC-BCWPcum 

Index 


(2) 


Use  of  different  performance  indices  results  in  different  EAC  forecasts.  The 
two  indices  used  primarily  in  the  DoD  (but  not  mandated)  are  cost  performance 
index  and  composite  index.  Use  of  these  indices  results  in  the  EACcpi,  referred  to  as 
the  “best  case”  EAC,  and  the  EACcomposite,  which  is  called  the  “worst  case”  EAC 
(“DAU  Gold  Card-July  2012,”  2012).  These  EAC  values  are  reported  as  best  and 
worst  case  EACs  in  the  monthly  DAES  reports. 


(1)  Best  Case  EAC.  The  best  case  approach  for  calculating  EAC  involves 
using  the  current  CPI  of  the  program  as  the  performance  index.  It  assumes  that  the 
rest  of  the  work  remaining  will  be  done  according  to  the  same  cost  efficiency 
recorded  to  date.  Equation  3  depicts  this  relationship: 


ETC, 


CPI 


BAC  -  BCWP,um 
CPI 


(3) 


Substituting  ETCCPI  into  Equation  1  results  in  the  EACcpi  expression  in 
Equation  4. 


EACcpi 


=  ACWP  + 


BAC  -  BCWP^^rri 
CPI 


(4) 


(2)  Worst  Case  EAC.  The  worst  case  approach  for  calculating  EAC  involves 
using  a  composite  performance  index  called  the  schedule  cost  index  (SCI).  Equation 
5  shows  the  SCI  expression. 


SCI  =  SPI  X  CPI 


(5) 


It  assumes  that  the  rest  of  the  future  work  will  follow  the  cost  efficiency 
determined  by  the  cost  performance  index  (CPI),  as  well  as  the  schedule  efficiency 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


determined  by  the  scheduled  performance  index,  generating  the  SCI.  Equation  6 
depicts  this  relationship: 


ETC  = 


BAC  -  BCWP^urn 
SCI 


(6) 


This  approach  incorporates  a  tendency  for  programs  to  perform  with  CPIs 
and  SPIs  less  than  one  (an  indication  of  inefficiency).  The  product  of  two  indices 
less  than  one  has  the  compounding  effect  of  raising  the  ETC  more  than  using  the 
CPI  alone  and  consequently  results  in  a  higher  EAC  forecasts  than  the  best  case 
approach  (see  Equation  7). 


EAC  =  ACWP  + 


BAC  -  CDWP,^rn 
SCI 


(7) 


This  study  uses  the  earned  value  data  reported  in  the  Defense  Acquisition 
Executive  Summaries  (DAES)  on  the  government’s  Defense  Acquisition 
Management  Information  Retrieval  (DAMIR)  website.  DAMIR  is  a  DoD  initiative 
that  streamlines  acquisition  management  and  oversight  by  leveraging  numerous 
government  databases  into  one  central  repository.  The  DAMIR  databases  used  in 
this  study  were  the  Selected  Acquisition  Report  (SAR)  and  the  Services’  Defense 
Acquisition  Executive  Summary  (DAES).  The  DAES  information  contains  some  of 
the  earned  value  data  from  the  contract  performance  reports  (DD  Forms  2734/1-5, 
APR  2005)  on  a  nearly  monthly  basis  (not  all  EV  data  available  in  the  CPR  formats 
1-5  is  available  for  study).  The  SARs,  in  the  most  extreme  case,  are  published 
three  times  per  year,  but  more  typically  are  published  once  annually  and  do  not 
contain  sufficient  EV  data  for  meaningful  analysis  in  this  study;  however,  the  SAR  is 
useful  because  it  provides  background  information  on  each  program  and  some 
explanations  of  program  actions  taken.  The  analysis  reported  in  this  thesis  is  based 
almost  exclusively  on  the  DAES  data  because  of  the  frequency  and  quality  of  the  EV 
data  provided  in  DAES. 


Program  Selection 


Since  2001 ,  12  MDAPs  were  cancelled  before  they  could  field  an  operational 
system  (Harrison,  2011).  These  programs  are  listed  in  Table  1 .  Of  the  12  cancelled 
programs,  DAMIR  earned  value  data  was  available  for  the  eight  programs 
annotated  with  an  asterisk.  These  eight  programs  made  up  the  cancelled  programs 
sample  of  this  study. 
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Table  1.  Major  Programs  Cancelled  Since  2001  Without  Fielding  Any 

Operational  Systems 

(adapted  from  Harrison,  2011) 


Program  Name 

Sunk  Cost 

(in  billions  of  then-year  dollars) 

Source 

Fuwre  Combat  Systems  (PCS)* 

S18.1  B 

September  2008  S.AR 

Comanche* 

S7.9B 

December  2003  S.AR 

National  Polar-orbiting  Operational  Emironmental 
Satellite  Svstem  (NTOESS)* 

S5.SB 

December  2009  S.AR 

Mi-?!  Presidential  HeUcopter* 

S3.7B 

September  2008  S.AR 

Expeditionar\'  Fighting  S'ehicle  (EFS”)* 

S3.3B 

December  2010  S.AR 

Transformational  S.A.TCOM  (TS.-XT) 

s3.:b 

.Annual  Budget  Justification  Document 

Crusader* 

S2.:b 

December  2001  S.AR 

.A.£h'anced  SE.AL  Deli\'er>'  System  (.A.SDS) 

S0.6B 

December  2005  S.AR 

-Armed  Reconnaissance  Helicopter  (.ARH)* 

S0-5B 

September  2008  S.AR 

Aerial  Common  Sensor  (ACS)* 

S0.4B 

December  2005  S.AR 

CG{x)  Next  Generation  Cruiser 

so.;b 

.Annual  Budget  Justification  Document 

CS.AR-X 

S0.2B 

.Annual  Budget  Justification  Document 

*  Program  used  in  this  study 

Conceivably,  more  can  be  learned  by  comparing  cancelled  programs  to 
fellow  troubled  programs  than  by  comparison  to  on-track  programs.  One  would 
expect  the  on-track  programs  to  outperform  the  cancelled  programs  in  most,  if  not 
all,  earned  value  performance  metrics.  For  this  study  we  define  a  troubled  program 
as  a  program  that  experienced  consecutive  unfavorable  (negative)  cumulative  cost 
or  schedule  variance  percentages  greater  than  10%.  The  selection  criterion  used 
for  the  comparison  sample  of  the  non-cancelled  programs  was  three-fold:  (1)  the 
program  was  not  cancelled,  (2)  earned  value  data  was  available  for  the  major 
program,  and  (3)  the  program  had  to  be  troubled  (as  defined  above). 

Eight  programs  were  selected  from  the  90  active  programs  in  DAMIR  for 
use  as  a  control  group.  The  programs  are  organized  by  service  on  DAMIR  and 
then  alphabetized  within  the  service.  The  first  row  of  each  service  was  analyzed 
until  eight  programs  that  met  the  criteria  were  obtained.  The  first  eight  non- 
cancelled  programs  that  met  the  criteria  for  troubled  programs  were  chosen  as  the 
comparison  sample  of  non-cancelled  programs.  These  programs  are  listed  Table  2. 
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Table  2.  Comparison  Sample  of  Troubled  Non-Cancelled  Major  Programs 

(Hodgson,  2013) 


Program  Name _ 

HI  Upgrades 

Joint  Precision  Approach  and  Landing  System  (JPALS) 
CH-53K 

Global  Positioning  System  (GPS)  III 
Joint  High  Speed  \’e5sel  (JHS\') 

PATRIOT  Advanced  Capabilit>'-3  (PAC-3)  System 
UTi-dOM 

Joint  Primary'  Aircraft  Training  System  tJPATS) _ 


This  study  hypothesized  that  the  variables  presented  in  Table  3  were  the 
most  likely  differentiators  of  cancelled  programs.  In  order  to  sufficiently  analyze 
the  hypotheses  that  these  variables  are  different  in  cancelled  programs,  the  following 
EV  data  was  collected  for  each  program: 

•  Cost  variance  percentage 

•  Schedule  variance  percentage 

•  Program  Manager’s  Estimate  at  Completion  (PM  EAC) 

•  Contractor’s  Estimate  at  Completion 

•  Cost  variance  resets 

•  Schedule  variance  resets 
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Table  3.  Variables  Used  in  Analysis 

(Hodgson.  2013) 


Independent  Variables _ 

Cost  \'ariance  ®o 

Cost  \*ariance  °o  at  25° o  completion 

Cost  \'ariance  °o  at  50°  o  completion _ 

Schedule  \’ariance  °o 

Schedule  \*ariance  °o  at  25°  o  completion 

Schedule  \'ariance  °o  at  50°  o  completion _ 

Cost  Gro\\th  (PM  Estimate  at  Completion) 

Cost  Growth  (PM  Estimate  at  Completion)  at  25°  o  completion 

Cost  Gro\\'th  (PM  Estimate  at  Completion)  at  50°  o  completion _ 

Cost  Gro\\th  (Contractor  Estimate  at  Completion) 

Cost  Gro\%'th  (Contractor  Estimate  at  Completion)  at  25°o  completion 
Cost  Gro%\'th  (Contractor  Estimate  at  Completion)  at  50°o  completion 
Difference  benveen  Contractor  and  Program  Manager's  EAC 
Difference  benveen  Contractor  and  Program  Manager's  EAC  (0-25°  o) 
Difference  benveen  Contractor  and  Program  Manager's  EAC  (26-50°  o) 
Difference  benveen  Contractor  and  Program  Manager's  EAC  (51-75°o) 
Cost  \'ariance  Reset  Frequency 

Schedule  \'ariance  Reset  Frequency _ 


Contract  Selection 

All  MDAPs  have  multiple  contracts  for  each  phase  of  the  weapons  systems 
development  to  handle  different  components  and  functions  of  systems 
acquisition.  To  maintain  consistency  between  the  samples,  the  largest 
Engineering  and  Manufacturing  Development  phase  contract  was  used  for  each 
program.  The  available  raw  data  for  the  variables  for  each  program’s  largest  EMD 
contract  was  extracted  from  the  DAMIR  site  under  the  “Earned  Value”  tab  using  the 
“Cumulative”  and  “Summary”  reports.  While  the  DAES  reports  are  missing  some 
data,  sufficient  data  existed  for  each  program  to  conduct  the  analysis. 

Cost  Variance  Percentage 

The  cost  variance  percentages  were  extracted  from  every  CPR  available  in 
the  DAES  reports.  Hodgson  (2013)  calculated  the  mean  and  medians  presented  in 
Table  4.  We  note  that  the  overall  median  and  mean  CV%  of  non-cancelled  programs 
is  more  unfavorable  than  the  cancelled  programs,  which  suggests  that  the  non- 
cancelled  programs  are  at  least  as  “troubled”  as  their  cancelled  counterpart. 
Additionally,  it  suggests  that  CV%  may  not  be  a  discriminating  variable  in  a 
program’s  survival. 
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Table  4.  CV  Percentage  Means  and  Medians  of  All  Programs  in  Study 

(Hodgson,  2013) 


Cancelled  prcrrani 

Median  C\'’o 

AleanCV’o 

Std.  Dev. 

Xon-cancelled  proeram 

Median  C\’°e 

.Mean  C\'“o 

Std.  Dev. 

\'K-'l 

o.i: 

-1.31 

-i."5 

K1  Upsrades 

-0.9" 

-2.66 

'.16 

PCS 

0.5S 

0.63 

1.38 

.T.ALS 

-6.28 

-6.99 

2.54 

Comanche 

-1.59 

-4.31 

3.45 

CK-53:< 

-0.9" 

-1.50 

4.02 

XPOESS 

-3.25 

-4.58 

4.90 

GPSM 

-5.62 

4.82 

EFC 

-6.93 

-'.14 

3.44 

-TKSA' 

-24.3" 

-24.64 

13.09 

Crusader 

-1.94 

-3.03 

3.31 

PAC-3 

-5.5-1 

-8.20 

3.58 

-Armed  Recon  Kelo 

-4.5S 

-S.‘3 

9.58 

UK-60M 

-5.-" 

-4.26 

4.03 

.ACS 

-1.94 

-5.4" 

11.05 

.TATS 

-9.54 

-8.13 

12.50 

CK-etall 

-2.59 

•  ♦ 

CK-erall 

-5.58 

■8.02 

Hodgson  (201 3)  used  the  same  CV%  data  to  examine  the  impact  of  the 
timing  of  cost  variance  percentages  on  program  cancellation  at  the  25%  and  50% 
completion  points  of  the  contract.  For  these  two  completion  points,  the  average  of 
the  CV%  within  +/-  5%  of  the  respective  completion  points  was  used  for  the  CV%  of 
each  program.  Tables  5  and  6  contain  the  CV%  at  these  completion  points  for  the 
sampled  cancelled  and  non-cancelled  programs  (note:  data  for  VH-71  and  ACS 
were  not  available  in  DAES  for  these  contract  completion  points). 

Table  5.  Cancelled  Programs’  CV  Percentages  at  25%  and  50%  Contract 

Completion  Points 

(Hodgson,  2013) 


Cancelled  program 

CV®o  at  25°o  Contract 
Completion  Point 

C\'®o  at  50°  0  Contract 
Completion  Point 

\'H-71 

-13.01 

PCS 

1.12 

-1.07 

Comanche 

-9.07 

-4.96 

XPOESS 

1.11 

-1.34 

EF\' 

-13.19 

-4.31 

Crusader 

-5.47 

-6.16 

Armed  Recon  Helo 

-6.18 

-21.37 

-ACS 

Median 

-5.82 

-4.96 

Mean 

-5.28 

-7.46 
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Table  6.  Non-Cancelled  Programs’  CV  Percentages  at  25%  and  50% 

Contract  Completion  Points 

(Hodgson,  2013) 


Xon-cancelled  program 

C\’®o  at  25°o  Contract 
Completion  Point 

CV®o  at  50®  0  Contract 
Completion  Point 

HI  Upgrades 

-11.13 

-9.78 

JP.ALS 

-4.30 

-6.19 

CH-53K 

-1.23 

-5.16 

GPS  III 

-4.55 

-12.49 

JHS\* 

-7.78 

-10.96 

PAC-3 

-6.33 

-8.15 

UH-60M 

7.84 

-1.51 

JPATS 

14.85 

-11.29 

Median 

-4.42 

-8.96 

Mean 

-1.58 

-8.19 

On  average,  cancelled  programs  in  the  study  had  more  unfavorable  CV%  at 
the  later  stages  of  the  contract;  however,  the  median  CV%  appears  to  improve  from 
25%  to  50%  completion  point.  Table  6  shows  that  both  the  mean  and  median  CV% 
becomes  more  unfavorable  over  time  for  the  sampled  non-cancelled  programs. 
When  comparing  the  mean  CV%  between  cancelled  and  non-cancelled  programs  in 
Tables  5  and  6,  cancelled  programs  maintain  worse  CV%  at  the  25%  completion 
point,  but  slightly  more  favorable  at  the  50%  completion  point.  These  observations 
are  not  intuitive  and  make  drawing  any  initial  conclusions  difficult. 

Schedule  Variance  Percentage 

The  schedule  variance  percentages  were  extracted  and  compiled  from  every 
available  CPR  in  the  DAES  reports.  Hodgson  (2013)  calculated  the  mean  and 
median  for  each  program  presented  in  Table  7.  The  mean  SV  percentage  is  slightly 
more  unfavorable  (more  negative)  in  cancelled  programs  while  the  cancelled 
programs’  median  SV%  is  slightly  less  unfavorable. 
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Table  7.  SV  Percentage  Means  and  Medians  of  All  Programs  in  Study 

(Hodgson,  2013) 


Cancelled  program 

Median  SV°o 

MeanSV’o 

Std.  Dev. 

Xon-cancelled  program 

Median  SV°o 

Mean  SV’o 

Std.  Dev. 

\li-ll 

-1.94 

-2.20 

1.13 

HI  Upgrades 

-1.24 

-2.67 

2.74 

FCS 

-0.91 

-0.94 

0.45 

IP.ALS 

-2.31 

-2.31 

1.14 

Comanche 

-2.19 

-2.10 

1.75 

CH-53K 

-2.73 

-3.16 

2.59 

XPOESS 

-1.01 

-4.44 

6.79 

GPS  III 

-3.41 

-3.76 

2.47 

EF\’ 

-3.71 

-3.82 

1.74 

ms\' 

-6.07 

-6.05 

2.47 

Crusader 

-0.9S 

-3.24 

4.72 

PAC-3 

-1.85 

-3.33 

4.28 

.Armed  Recon  Helo 

-19.78 

-16.30 

11.89 

UH-60M 

-7.86 

-8.70 

4.35 

.ACS 

-5.86 

-9.93 

10.44 

IPATS 

-6.23 

-6.19 

5.69 

Overall 

-2.06 

-5.37 

Overall 

-3.07 

-4.52 

To  analyze  the  timing  of  schedule  variances  and  their  potential  effect  on 
cancellation,  Hodgson  (2013)  used  the  same  SV%  data.  For  the  25%  and  50% 
completion  points  of  the  contract,  the  average  of  the  SV%  within  +/-  5%  of  these 
completion  points  was  used  for  the  SV%  of  each  program.  Tables  8  and  9  contain 
the  SV%  at  these  completion  points  for  the  sampled  cancelled  and  non-cancelled 
program. 

Table  8.  Cancelled  Programs’  SV  Percentages  at  25%  and  50%  Contract 

Completion  Points 

(Hodgson,  2013) 


Cancelled  program 

S\'®o  at  25®  0  Contract 
Completion  Point 

S\'®o  at  50°  0  Contract 
Completion  Point 

FCS 

-0.93 

-0.93 

Comanche 

1 

1 

o 

XPOESS 

-5.56 

-0.36 

EF\* 

-6.80 

-4.44 

Crusader 

-10.51 

-11.39 

-Armed  Recon  Helo 

.4CS 

-30.98 

-20.71 

Median 

-6.18 

-4.44 

Mean 

-9.83 

-7.56 
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Table  9.  Non-Cancelled  Programs’  SV  Percentages  at  25%  and  50% 

Contract  Completion  Points 

(Hodgson,  2013) 


Non-cancelled  program 

S\'®o  at  25°  0  Contract 
Completion  Point 

S\'°o  at  50°  0  Contract 
Completion  Point 

HI  Upgrades 

-3.68 

-3.08 

JP.ALS 

-3.31 

-3.41 

CH-53K 

-1.52 

-0.55 

GPS  III 

-1.72 

-6.03 

JHS\' 

-8.99 

-5.49 

PAC-3 

-16.13 

-5.32 

L'H-dOM 

-25.63 

-7.22 

JPATS 

-6.10 

-6.77 

Median 

-4.89 

-5.41 

Mean 

-8.38 

-4.73 

Table  8  suggests  that  on  average  cancelled  programs  have  more  favorable 
SV%  at  the  later  stages  of  the  contract.  Table  9  shows  that  while  the  mean  SV% 
seems  to  improve  over  time,  the  median  SV%  becomes  more  unfavorable  for  the 
sampled  non-cancelled  programs.  When  comparing  the  mean  and  median  SV% 
between  cancelled  and  non-cancelled  programs  in  Tables  9  and  10,  cancelled 
programs  maintain  worse  SV%  at  the  25%  completion  point.  There  is  conflicting 
evidence  between  the  mean  and  median  SV%  at  the  50%  point.  Again,  these 
general  observations  based  on  very  small  sample  sizes  make  drawing  initial 
conclusions  challenging. 

Cost  Growth 

To  analyze  cost  growth  as  a  potential  discriminating  variable,  Hodgson 
(2013)  recorded  the  program  manager  estimate  at  completion  and  contractor  EACs 
for  every  CPR.  Initially  the  cumulative  cost  growth,  that  is,  the  difference  between 
the  EAC  at  the  beginning  of  the  contract  and  the  current  EAC  would  seem  the 
logical  metric.  However,  due  to  changes  in  the  program  baseline  (“re-baselining”) 
during  the  course  of  the  contract,  the  cumulative  cost  growth  did  not  prove  a 
consistent  measure  from  program  to  program.  When  a  program  experiences 
significant  cost  growth  or  schedule  delays,  it  is  not  uncommon  for  the  program 
manager  to  request  that  it  be  “re-baselined.”  This  gives  the  program  a  fresh  start, 
and  leads  to  a  new  “beginning”  EAC.  One  possible  way  to  handle  the  effect  of  re¬ 
baselining  is  to  consider  the  re-baselined  program  a  new  program.  This  method  was 
discarded  because  it  was  difficult  to  determine  in  some  cases  when  or  if  a  program 
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had  been  re-baselined  (and  in  some  cases  the  EAC  would  go  down  without  any 
evidence  of  re-baselining). 

Alternatively,  it  is  possible  to  ignore  the  re-baselined  EAC  and  continue  to 
calculate  the  difference  between  the  original  EAC  and  the  current  EAC.  Doing  so  is 
problematic  however,  as  it  fails  to  recognize  that  some  programs  are  re-baselined 
due  to  changes  in  scope  or  requirements,  so  that  the  observed  cost  growth  is  not 
indicative  of  program  health,  but  merely  a  reflection  of  legitimate  changes  in  program 
scope.  Since  the  objective  of  this  research  was  to  find  variables  that  may  predict 
program  cancellation,  measuring  the  difference  between  the  original  EAC  and  the 
new  EAC  could  be  misleading. 

To  address  the  difficulties  mentioned  above,  a  cost  growth  metric  was 
constructed  by  Hodgson  (2013)  to  facilitate  comparison  of  programs.  The  purpose  of 
the  metric  is  to  measure  the  cumulative  effect  of  marginal  changes  in  the  EAC.  This 
metric  has  the  advantage  that  it  is  less  distorted  by  re-baselining  yet  still  captures 
the  overall  effect  of  increases  in  the  EAC  from  the  beginning  of  the  program  to  the 
current  period.  The  metric  for  both  PM  EAC  and  contractor  EAC  was  calculated 
using  Equation  8: 


(8) 


EAC  Growth% 


—  EAC  Growth%past  + 


'present 


This  constructed  metric  provides  a  conservative  estimate  of  the  cost  growth 
experienced  by  a  program,  while  eliminating  the  problems  caused  by  re-baselining. 
This  metric  is  hereafter  referred  to  as  “cost  growth.”  Note  that  the  cumulative  cost 
growth  percentage  (computed  by  taking  the  difference  between  the  current  EAC  and 
the  original  EAC  and  dividing  the  difference  by  the  original  EAC)  will  always  be 
higher  than  the  “cost  growth”  metric. 

For  the  25%  and  50%  contract  completion  points,  the  average  cost  growths 
within  +!-  5%  of  25%  and  50%  completion  points,  respectively,  were  computed  by 
Hodgson  (2013)  and  used  for  the  cost  growth  of  each  program.  The  total  cost 
growths  and  cost  growths  at  25%  and  50%  completion  points  were  calculated  for  all 
programs  in  the  study  and  are  presented  in  Tables  10  through  13.  Where  data  is 
missing,  it  is  because  it  was  unavailable  in  DAES. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-  17- 


Table  10.  Cost  Growth  for  Cancelled  Programs  (PM  EAC) 

(Hodgson,  2013) 


Cancelled  program 

Total  Cost  Growth 
IPME.AC) 

at  25°  0  Contract 
Completion  Point 

at  50°  0  Contract 
Completion  Point 

\'H-71 

-36,64°  0 

FCS 

55.28% 

41.11% 

47.23°o 

Comanche 

93.48°o 

55.81°o 

XPOESS 

113.52°o 

25.13°o 

88.25°o 

EF\' 

86.31°o 

6.16°o 

23.15°o 

Crusader 

399.29°o 

337.04°o 

258.43°  0 

.Armed  Recon  Helo 

151.21°o 

-0.51°o 

23.92°o 

.ACS 

22.31°o 

Median 

89.90°o 

33.12°o 

47.23°o 

Mean 

110.60°o 

77.46°o 

88.19°o 

Table  11.  Cost  Growth  for  Non-Cancelled  Programs  (PM  EAC) 

(Hodgson,  2013) 


Xon-cancelled  program 

Total  Cost  Growth 
(PME.AC) 

at  25°  0  Contract 
Completion  Point 

at  50°  0  Contract 
Completion  Point 

HI  Upgrades 

109.49°o 

20.99°  0 

45.93°o 

JP.ALS 

20.00°  0 

0.72°o 

2.86°o 

CH-53K 

33.03°o 

0.00°o 

0.01°o 

GPS  III 

-2.01°o 

0.00°o 

-0.04°  0 

JHSA' 

27.17°o 

2.74°o 

2.74° 0 

P.AC-3 

67.63°o 

-4.88° 0 

o 

o 

o 

o 

UH-60M 

83.12°o 

0.00°o 

70.81°o 

JP.ATS 

58.51°o 

30.62°  0 

66.56°  0 

Median 

45.77°o 

0.36°o 

16.43°o 

Mean 

49.62°  0 

6.27°o 

27.36°o 
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Table  12.  Cost  Growth  for  Cancelled  Programs  (Contractor  EAC) 

(Hodgson,  2013) 


Cancelled  program 

Total  Cost  Growth 
(Contractor  E.AC) 

at  25°  0  Contract 
Completion  Point 

at  50°  0  Contract 
Completion  Point 

SH-Tl 

20.04°  0 

PCS 

55.30°o 

41.28°o 

50.04°  0 

Comanche 

98.81°o 

55.96°o 

XPOESS 

118.13°o 

24.02° 0 

92.87°o 

EF\' 

84.54°  0 

3.08°o 

27.86°o 

Crusader 

393.44°o 

329.59°  0 

338.79°o 

-Armed  Recon  Helo 

143.95°o 

-0.51°o 

52.83°  0 

-ACS 

21.60°o 

Median 

91.68°o 

32.65°o 

52.83°  0 

Mean 

116.98°o 

75.57°o 

112.48°o 

Table  13.  Cost  Growth  for  Non-Cancelled  Programs  (Contractor  EAC) 


Xon-cancelled  program 

Total  Cost  Growth  (Contractor 
E.AC) 

at  25°  0  Contract 
Completion  Point 

at  50° 0  Contract 
Completion  Point 

HI  Upgrades 

106.23°  0 

19.49°o 

43.27°o 

JP.ALS 

24.28°o 

2.45°o 

9.74°o 

CH-53K 

37.65°o 

21.83°o 

32.14°o 

GPS  III 

0.60°  0 

3.92°o 

17.82°o 

JHSV 

27.05°  0 

0.00°  0 

2.06°o 

P.AC-3 

66.71°o 

-9.20°  0 

22.86°o 

UH-60M 

5.00°  0 

0.00°  0 

0.00°o 

JP.ATS 

56.99°o 

5.13°o 

102.77°o 

Median 

32.35°o 

3.19°o 

20.34°o 

Mean 

40.56°  0 

5.45°  0 

28.83°o 

We  can  make  some  general  observations  from  Tables  1 1  through  14.  As 
expected,  regardless  of  estimate  type,  the  cost  growth  metric  increases  as  programs 
progress.  In  addition,  cost  growth  appears  to  be  significantly  higher  in  cancelled 
programs  than  in  the  non-cancelled  counterparts.  Moreover,  the  data  appears  to 
show  that  the  there  are  fewer  differences  in  program  manager  and  contractor 
estimates  in  non-cancelled  programs  and  that  the  greatest  difference  in  estimates 
appears  to  occur  at  the  50%  completion  point  of  cancelled  programs. 

Differences  in  the  Estimates  at  Completion 

To  investigate  the  potential  effect  the  differences  between  the  two  estimates 
may  have  on  cancellation,  Hodgson  (2013)  first  normalized  the  differences.  The 
calculation  to  normalize  the  differences  is  based  on  the  PM  EAC  and  Contractor 
EAC  values  reported  in  each  program’s  CPR  in  the  DAES  report.  To  normalize  the 
differences  for  program  comparison,  the  magnitude  of  the  difference  is  divided  by  its 
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corresponding  program  manager’s  estimate  at  completion  for  each  program.  For 
example,  if  the  program  manager’s  estimate  at  completion  was  $5,000  and  the 
contractor’s  estimate  at  completion  was  $4,500  at  a  certain  contract  completion 
percentage,  the  difference  between  the  two  estimates  is  $500  and  the  normalized 
difference  is  $500/$5,000  or  0.10. 

The  data  were  organized  into  the  following  four  periods  to  study  the  potential 
effects  of  the  timing  of  the  difference:  (1)  total  normalized  difference,  (2) 
normalized  difference  from  0-25%  contract  completion  point,  (3)  normalized 
difference  from  26-50%  contract  completion  point,  and  (4)  normalized  difference 
from  51-75%  contract  completion  point.  The  median  total  EAC  difference  and 
averages  of  the  three  periods’  EACs  were  calculated  for  each  program  in  the  study 
and  are  presented  in  Tables  14  and  15.  The  cells  of  the  table  that  remain  empty  did 
not  have  sufficient  data  for  analysis. 

Table  14.  Normalized  Differences  in  Program  Manager  and  Contractor  EAC 

for  Cancelled  Programs 

(Hodgson,  2013) 


Cancefled  program 

Total  EAC 

Difference* 

EAC  Difference 
(from  0-25“  o) 

EAC  Difference 

(from  26-50“  o) 

EAC  Difference 

(from  51 -"6“  o) 

38.3  l^o 

32.16“o 

FCS 

0.00“  0 

0.00“o 

0.00“o 

0.00“o 

Comanche 

3,""“o 

4.99“o 

5.8‘“o 

NPOESS 

3.0-“ 0 

18.69“o 

12.01“o 

3,13“o 

EFV  SDDl 

1.39“o 

1.16“o 

2.51“o 

2.04“o 

Crusader 

l.:8“o 

1.19“o 

1.49“o 

Armed  Recon  Flelo 

0.00“  0 

0.00“o 

0.61“o 

2.68“o 

ACS 

:.80“o 

5.19“o 

Median 

:.09“o 

3.08“o 

1.85“o 

2.36“o 

Mean 

6.33“  0 

5.01“o 

3."0“o 

6.92“o 

•  Median  EAC  Difference 
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Table  15.  Normalized  Differences  in  Program  Manager  and  Contractor  EAC 

for  Non-Cancelled  Programs 

(Hodgson,  2013) 


Xon-cancelled  program 

Total  EAC 

Difference* 

EAC  Difference 
(from  O-25O0) 

E.4C  Difference 
(from  26-500 0) 

EAC  Difference 
(from  51 -"60  0) 

HI  Upgrades 

OT'Oo 

O.I500 

I.5200 

5.3400 

JP.4LS 

T.rOOo 

3.OOO0 

4.4900 

3.IIO0 

CH-53K 

6.460  0 

9.360o 

6.3IO0 

GPS  III 

8.8200 

9.6*Oo 

9.5500 

9.IIO0 

JHS\’ 

O.I3O0 

O.OQOo 

4.5IO0 

I.O2O0 

PACO 

0.8"0o 

O.OQOo 

I.2300 

4.6300 

UH-60M 

4,6'Oo 

O.OQOo 

I.I6O0 

6.3400 

JPATS 

5.3"Oo 

8.3^0, 

11.5600 

2.3IO0 

Median 

3.5IO0 

1.5'Oo 

4.5OO0 

4.980o 

Mean 

3."80o 

3.450  0 

5.4200 

4.'-Oo 

•  Median  E.4C  Dfference 

The  medians  and  means  tell  two  different  stories  in  this  case.  A  comparison 
of  the  medians  reveals  that  the  EAC  difference  is  worse  (greater)  in  non-cancelled 
programs  for  all  but  the  0-25%  group,  whereas,  a  comparison  of  the  means 
suggests  that  cancelled  programs  have  worse  EAC  differences  in  all  groups  but  the 
26-50%.  Conclusions  cannot  be  made  from  simply  comparing  the  means  and 
medians  in  this  way,  but  it  is  useful  in  providing  a  general  sense  of  how  the 
difference  in  estimates  may  effect  cancellation. 

Methodology 

To  understand  how  cancelled  programs,  and  non-cancelled  but  troubled 
programs  differ  from  each  other,  we  examine  differences  in  earned  value  metrics. 
The  challenge  we  face  is  that  the  small  number  of  programs  limits  our  sample  size 
and  our  ability  to  use  parametric  methods.  Using  methodology  that  works  for  small 
sample  sizes,  Hodgson  (2013)  examined  whether  cancelled  and  non-cancelled 
programs  are  different  on  average  in  terms  of  their  earned  value  metrics.  We  look  at 
whether  we  can  use  earned  value  metrics  to  predict  the  probability  of  failure,  and  we 
look  at  whether  over  time,  the  risk  of  program  failure  changes. 

Building  on  the  differences  observed  in  Mann-Whitney  by  Hodgson  (2013), 
we  want  to  estimate  the  probability  of  program  cancellation  based  on  earned  value 
metrics.  In  most  applications  where  the  dependent  variable  is  binary,  maximum 
likelihood  techniques  such  as  Logit  or  Probit  are  used  instead  of  OLS  because  they 
constrain  the  outcome  probability  to  be  less  than  1 00%  and  more  than  0%,  where 
linear  models  can  estimate  probabilities  outside  these  bounds.  Probit  and  logit 
better  handle  dichotomous  variables  by  using  an  iterative  optimization  routine  to 
maximize  a  log  likelihood  function  (Kennedy,  2003). 
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Unfortunately,  maximum  likelihood  estimators  like  logit  are  generally 
unreliable  when  samples  are  small  (Statistical  Consulting  Group,  2014).  Maximum 
likelihood  estimators  suffers  from  bias  when  sample  sizes  are  small  or  the  ratio  of  Is 
to  Os  is  skewed,  a  tendency  exacerbated  by  the  inclusion  of  many  explanatory 
variables  (Long,  1997).  Exact  logit  is  the  prescribed  solution  in  the  case  of  small 
sample  sizes,  where  traditional  maximum  likelihood  estimation  is  undesirable  as  it 
produces  inconsistent  results  (Hart  &  Clark,  1999).  As  we  only  have  16 
observations,  in  addition  to  exact  logistic  we  limit  the  inclusion  of  explanatory 
variables  to  include  them  one  at  time.  Exact  logistic  returns  the  log  odds  of  the 
outcome  variable  as  a  linear  model  of  the  predictor  variables.  We  can  interpret  the 
coefficients  as  odds  ratios  meaning  the  coefficient  is  essentially  a  multiplier. 

Secondly,  we  consider  program  survival  using  parametric  survival  analysis. 
We  can  think  of  program  survival  as  sick  patient  survival  over  time  after  treatment. 

In  this  analysis  we  look  at  time  in  four  periods — between  0-25%  complete,  25%- 
50%  complete,  50%-75%  complete,  and  75-100%  complete  to  evaluate  the  impact 
of  program  development  and  earned  value  metrics  over  the  program’s  development 
on  the  probability  of  cancellation.  We  model  this  parametrically  with  proportional 
hazard  functions  using  an  exponential  model,  which  keeps  time  neutral. 

We  cannot  use  all  variables  together  in  the  same  models — schedule  variance 
is  highly  correlated  with  cost  variance,  and  estimate  at  completion  growth  of  both 
contractors  and  program  managers  are  too  highly  correlated  with  each  other. 
Schedule  variance  is  also  too  highly  correlated  with  both  EAC  growths  figures.  Thus 
we  break  down  our  analysis  into  single  variable  models  and  then  add  cost  variance 
to  the  EAC  at  completion  variance  models  as  a  control 

Results 

Exact  Logistics 

Using  exact  logistic  we  measured  the  impact  of  seven  EV  metrics  on  the 
probability  of  troubled  program  cancellation  testing  each  variable  separately.  The 
results  are  shown  in  Table  1 6.  We  show  the  coefficients  for  each  model  along  with 
the  standard  deviation  (in  parentheses).  Asterisks  indicate  significant  results.  The 
model  score  is  a  chi-squared  score  and  the  p  value  beneath  it  gives  the  overall 
significance  of  the  model.  Not  surprisingly,  as  these  are  univariate  models — the 
models  with  significant  p  are  also  the  models  with  a  significant  model  score. 
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Table  16.  Exact  Logistics  Models  for  EV  Metrics 


MODELS 

VARIABLES 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

Cost  Variance 

1.33* 

(0.084) 

Cost  Variance  Reset 

Frequency 

1.64 

(0.73) 

Schedule  Variance 

0.973 

(0.919) 

Schedule  Variance  Reset  Freq 

0.827 

(1) 

Cost  Growth  PM  EAC 

2.68 

(0.23) 

Cost  Growth  Contractor  EAC 

8.74* 

(0.05) 

Contractor  and  PM  EAC  Diff 

26.7 

(0.92) 

observations 

16 

16 

16 

16 

16 

16 

16 

model  score 

2.4 

0.48 

0.06 

0.09 

1.56 

2.65 

0.3 

p  value 

0.08 

0.73 

0.92 

1 

0.23 

0.05 

0.92 

The  coefficients  for  exact  logit  give  a  multiplier  for  the  likelihood  of  program 
cancellation.  We  find  that  the  cost  growth  based  on  the  contractor’s  estimate  is  a 
significant  indicator  of  program  cancellation.  Programs  with  one  hundred  percent 
cost  growth  are  8.74  times  more  likely  to  experience  program  cancellation.  In  other 
words,  a  1 0%  cost  growth  increases  the  chances  of  a  program  being  cancelled  by 
73%.  We  also  find  that  programs  with  high  but  positive  cost  variance  are  more  likely 
to  suffer  program  cancellation.  Specifically,  programs  with  100%  positive  cost 
variance  are  33%  more  likely  to  experience  a  cancellation.  Both  of  these  results  are 
statistically  significant  at  the  10%  level. 

Hazard  Function 

We  next  use  Hazard  functions  to  look  at  the  risks  to  programs  over  their 
maturity  from  their  EV  metrics.  The  results  are  shown  in  Table  1 7. 
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Table  17.  Hazard  Functions  for  EV  Metrics 


Modeis 

(1) 

(2) 

(3) 

(4) 

(5) 

(6) 

VARIABLES 

Schedule  Variance 

0.0222 

(0.0614) 

Cost  Variance 

0.0472 

0.0493 

0.0514 

(0.0410) 

(0.0462) 

(0.0458) 

Cost  Growth  Contractor  EAC 

0.540** 

0.535** 

(0.256) 

(0.255) 

Cost  Growth  PM  EAC 

0.511* 

0.502* 

(0.266) 

(0.265) 

Constant 

-1.718*** 

-1.598*** 

-2.179*** 

-2.230*** 

-1.941*** 

-1.976*** 

(0.459) 

(0.378) 

(0.463) 

(0.460) 

(0.477) 

(0.476) 

Observations 

50 

50 

46 

50 

46 

50 

Looking  at  the  hazard  functions,  we  find  that  cost  and  schedule  variance  are 
less  important  predictors  than  cost  growth  based  on  contractor  or  program  manager 
estimates  at  completion  which  are  significant  at  5%  and  10%  levels  respectively.  To 
understand  the  impact  of  any  factor  on  the  hazard  of  program  cancellation  we 
exponentiate  the  coefficient  which  gives  us  a  hazard  ratio  or  multiplier.  Here  a  100 
percentage  point  change  in  estimate  at  completion  leads  to  a  65%  to  75%  increase 
in  the  likelihood  of  cancellation,  or  a  10%  increase  in  the  estimate  would  lead  to  a 
6.5%  to  7.5%  increase  in  the  likelihood  of  program  cancellation  with  or  without 
controlling  for  cost  variance. 

Conclusions 

This  research  investigated  whether  there  are  differences  in  the  key  earned 
value  metrics  of  cancelled  and  troubled  non-  cancelled  programs,  and  whether 
these  metrics  can  help  predict  the  likelihood  of  program  cancellation.  Hodgson 
(2013)  found  that  cancelled  and  non-cancelled  programs  do  vary  across  their  EVM 
characteristics.  We  expanded  his  work  by  examining  the  data  with  two  more 
sophisticated  techniques,  exact  logit  and  hazard  analysis. 

Our  most  significant  finding  across  models  is  that  when  there  is  high  cost 
growth  in  the  EAC  reported  by  the  contractor,  programs  run  far  larger  risks  of 
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cancellation.  This  may  be  because  contractors  have  more  insight  into  their  cost 
structures  and  are  better  able  to  predict  EAC,  or  it  may  be  that  program  managers 
are  reluctant  to  increase  their  EAC  given  the  consequences  they  would  face  under 
the  Nunn-McCurdy  Act.®  We  find  less  robust  evidence  that  increases  in  PM 
estimates  and  high  cost  variance  also  can  drive  risk  of  program  cancellation. 

We  note  that  the  cost  growth  metric  (Equation  8)  was  developed  by  Hodgson 
(2013)  and  is  not  reported  as  EVM  data.  Given  the  significance  of  this  metric  in 
predicting  program  cancellation,  we  recommend  program  managers  consider  using 
this  metric  to  monitor  expected  cost  growth. 


®  The  Nunn-McCurdy  Act  (10  U.S.C.  §  2433)  requires  that  DoD  report  to  Congress  whenever  an 
MDAP  experiences  cost  overruns  that  exceed  certain  threshoids.  The  purpose  of  the  act  is  to  heip 
controi  cost  growth  in  major  defense  systems  by  hoiding  the  appropriate  Pentagon  officiais  and 
defense  contractors  pubiiciy  accountabie  and  responsibie  for  managing  costs. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-25- 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


References 


Abba,  W.  (1997).  Earned  value  management — Reconciling  government  and 
commercial  practices.  Program  Manager,  26,  58-63. 

Augustine,  N.  R.  (1997).  Augustine’s  laws  (6th  ed.).  Reston,  VA:  American  Institute 
of  Aeronautics  and  Astronautics. 

Berteau,  D.,  Ben-Ari,  G.,  Hofbauer,  J.,  Sanders,  G.,  Ellman,  J.,  &  Morrow,  D.  (2011). 
Cost  and  time  overruns  for  Major  Defense  Acquisition  Programs.  Washington, 
DC:  Center  for  Strategic  and  International  Studies. 

Capaccio,  T.  (2013,  April  22).  Pentagon  withholding  from  Lockheed  grows  over 
management.  Bloomberg.com.  Retrieved  from 

http://www.bloomberq.com/news  /2013-04-22/pentaqon-withholdinq-from- 

lockheed-qrows-over-  manaqement.html 

Charette,  R.  N.  (2008,  November  1).  What’s  wrong  with  weapons  acquisitions?  IEEE 
Spectrum.  Retrieved  from  http://spectrum.ieee.orq/aerospace/militarv/whats- 
wronq-with-  weapons-acquisitions 

Chen,  Y.,  &  Smith,  R.  (2001).  Equilibrium  cost  overruns.  Annals  of  Economics  and 
Finance,  2,  401-414. 

Christensen,  D.  D.  (1994,  Winter).  Cost  overrun  optimism:  Fact  or  fiction? 

Acquisition  Review  Quarterly,  25-38. 

Christensen,  D.  D.  (1999,  Summer).  Using  the  Earned  Value  Cost  Management 
Report  to  evaluate  the  contractor’s  estimate  at  completion.  Acquisition 
Review  Quarterly,  435. 

Christensen,  D.  S.  (1993,  March).  The  estimate  at  completion  problem:  A  review  of 
three  studies.  Project  Management  Journal,  24,  37-42. 

DAU  Gold  Card-July  2012.  (2012).  Retrieved  from  Defense  Acquisition  University 
website:  https ://acc.dau.mil/CommunitvBrowser.aspx?id=1 9577 

Defense  Acquisition  University  (DAU).  (2009).  Glossary  of  defense  acquisition 
acronyms  and  terms  (13th  ed.).  Ft.  Belvoir,  VA:  Defense  Acquisition 
University  Press. 

Defense  Acquisition  University  (DAU).  (2012a).  Defense  acquisition  guidebook. 
Retrieved  from  https://daq.dau.mil/Paqes/Default.aspx 

Defense  Acquisition  University  (DAU).  (2012b).  EVM  statutes,  regulatory  policy,  and 
DoD  implementation  guidance.  Retrieved  from  https://learn.dau.mil/ 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-27- 


Defense  Contract  Management  Agency.  (2006).  Earned  value  management 
implementation  guide.  Alexandria,  VA:  Author. 

Defense  Contract  Management  Agency.  (2012,  July  10).  Earned  Value  Management 
System  (EVMS)  program  analysis  pamphlet  (PAP)  (DCMA-EA  PAM  200.1 ). 
Fort  Lee,  VA:  Author. 

Dubos,  G.  F.,  Saleh,  J.  F.,  &  Braun,  R.  (2007,  September).  Technology  readiness 
level,  schedule  risk  and  slippage  in  spacecraft  design:  Data  analysis  and 
modeling.  Paper  presented  at  the  annual  meeting  for  the  American  Institute  of 
Aeronautics  and  Astronautics  Space,  Long  Beach,  CA. 

Eisner,  FI.  (2002).  Essentials  of  project  and  systems  engineering  management  (2nd 
ed.).  New  York,  NY:  John  Wiley  &  Sons. 

Estimate  at  completion  (EAC).  (2009).  In  Acquipedia.  Retrieved  from  Defense 
Acquisition  University  website: 

https://dap.dau. mil/acquipedia/Paqes/ArticleDetails.aspx?aid=684744f8- 

0de4-  4d1 7-a5f0-84533ad5841  a 


Government  Accountability  Office.  (2009).  GAO  cost  estimating  and  assessment 
guide  (GAO-09-3SP).  Retrieved  from 
http://www.qao.qov/new.items/d093sp.pdf 

Government  Accountability  Office.  (2014).  Defense  acquisitions:  Assessment  of 
selected  weapons  programs  (GAO-1 4-340SP).  Retrieved  from 
http://www.qao.qov/assets/670/662184.pdf 

Flarrison,  T.  (201 1 ).  Analysis  of  the  FY12  defense  budget.  Washington,  DC:  Center 
for  Strategic  and  Budgetary  Assessments. 

Flart,  R.  A.,  &  Clark,  D.  FI.  (1999).  Does  size  matter?  Exploring  the  small  sample 
properties  of  maximum  likelihood  estimation.  Retrieved  from 
http://www.polmeth.wustl.edu/media/Paper/hart99. pdf?oriqin=publication  det 

Ml 

Flodgson.  (201 3).  An  investigation  of  possible  discriminating  earned  value  variables 
in  Department  of  Defense  major  acquisition  program  cancellation  (Master’s 
thesis).  Monterey,  CA:  Naval  Postgraduate  School. 

Hosmer,  D.  W.,  &  Lemeshow,  S.  (1989).  Applied  logistic  regression.  New  York,  NY: 
John  Wiley  &  Sons. 

Kennedy,  P.  (2003).  Guide  to  Econometrics  5*^  Ed.  United  Kingdom:  MPG  Books 
Bodmin  Cornwall. 

Kerzner,  FI.  R.  (2009).  Project  management:  A  systems  approach  to  planning, 
scheduling,  and  controlling  (10th  ed.).  Floboken,  NJ:  John  Wiley  &  Sons. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY  -  28  - 

NAVAL  POSTGRADUATE  SCHOOL 


Long,  J.  S.  (1997).  Regression  models  for  categorical  and  limited  dependent 
variables.  Thousand  Oaks,  CA:  Sage  Publications. 

Makielski,  M.  (2009).  Examining  the  comprehensive  estimate-at-completion. 
Washington,  DC:  Defense  Contracting  Management  Agency. 

McKsymick,  E.  M.  (1995).  An  analysis  of  cost  overruns  in  the  development  of  the 
Navy’s  A-1 2  Avenger  aircraft  {Master’s  thesis).  Monterey,  CA:  Naval 
Postgraduate  School. 

Methods  Consultants  of  Ann  Arbor.  (2013).  What  is  the  difference  between  probit 
and  logit  models?  Retrieved  from 

http://www.methodsconsultants.com/tutorials/loqprob.html 

Morrison,  D.  C.  (1991,  March).  Deep-sixing  the  A-1 2.  Government  Executive,  23, 
30-35. 

Office  of  Management  and  Budget.  (2006,  June).  Capital  programming  guide  (0MB 
Circular  No.  A-1 1 ).  Washington,  DC:  Author. 

Office  of  the  Assistant  Secretary  of  Defense  for  Acquisition.  (201 2,  June  1 9). 

Integrated  Program  Management  Report  (IPMR)  data  item  description  (DID) 
[Memorandum].  Washington,  DC:  Author. 

Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  &  Logistics. 
(2007,  November  20).  The  defense  acquisition  system  (DoD  Directive 
5000.01).  Washington,  DC:  Author. 

Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  &  Logistics. 
(2008,  December  8).  Operation  of  the  defense  acquisition  system  (DoD 
Instruction  5000.02).  Washington,  DC:  Author. 

Performance  Assessments  and  Root  Cause  Analyses.  (2013).  Earned  Value 

Management  (EVM)  Division.  Retrieved  from  http://www.acq.osd.mil/evm/ 

Sipple,  V.,  White,  E.,  &  Greiner,  M.  (2004).  Surveying  cost  growth.  Ft.  Belvoir,  VA: 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and 
Logistics. 

Statistical  Consulting  Group,  University  of  California,  Los  Angeles.  (2014).  Stata 
data  analysis  examples:  Exact  logistic  regression.  Retrieved  from 
http://www.ats.ucla.edu/stat/sas/dae/exloqit.htm 

TutorialsPoint.  (2013).  Earned  value  management  tutorial.  Retrieved  from 
http://www.tutorialspoint.com/earn  value  manaqement 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-29- 


Tyson,  K.  W.,  Harmon,  B.  R.,  &  Utech,  D.  M.  (1994).  Understanding  costand 

schedule  growth  in  acquisition  programs  (IDA  Paper  P-2967).  Alexandria,  VA: 
Institute  for  Defense  Analyses. 

Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics.  (201 1 , 

January  24).  Earned  value  management  systems  performance,  oversight  and 
governance  [Memorandum].  Washington,  DC:  Author. 

Weapon  Systems  Acquisition  Reform  Act  of  2009,  Pub.  L.  No.  1 1 1  -23,  123  Stat. 
1704  (2009). 

Webb,  A.  (2003).  Using  earned  value:  A  program  manager’s  guide.  Burlington, 
VT:  Gower  Publishing. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-30- 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OE  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 

555  DYER  ROAD,  INGERSOLL  EIALL 

MONTEREY,  CA  93943 


www.acquisitionresearch.net 


